Basic formats of data

• string or character-based data
  • viewable/editable in a text processor
  • may consist of plain text, or specially encoded information (markup languages, tabular formats—“structured data”)
• image, audio, video, and sensor data
  • requires additional layers of software for viewing and editing
  • various encoding schemes depending on electronic source and desired use case
  • there is often a processing step to convert some of the information into structured data, for analysis
• compressed data (one of the above types, some bits removed)
  • compression may be “lossy” or “lossless”
  • decompression may occur on-the-fly (playing an MP3) or may need to be accomplished before consuming data (a zip file)

Ways to work with data

• Tabular format – quantitative, statistical methods (R’s preference)
• Document format (XML) – usually qualitative methods
• Computer vision and audition – algorithmic processing, deep learning models

Converting from XML or unstructured data to tabular format is popular, because it facilitates statistical analysis. An image containing

In this course, we are working with string data—interpreted as numeric values and pieces of text—in a tabular format.

What are R and RStudio?

What can you do with R and RStudio?

Example analyses

Here is a (very small) sampling of papers that use analytic methods which are available in R.

Why I prefer R

Our first R code

• When you open RStudio, you’ll see the console. This is where R waits for commands.
• Arithmetic operators: + - * / ^ (exponentiation) %% (modulus) %/% (integer division)
• As in algebra, an R function accepts arguments and returns a value.
• Common mathematical functions: sum(), mean(), median(), mode(), min(), max(), sd(), sqrt(), abs()
• Logarithms: log(x) for natural and log10(x) for base 10 (or log(x, base) for any base you want)
• \(e\) (Euler’s number) is represented with exp(1), where 1 is the desired exponent of \(e\).
• Rounding: round() for decimal places, signif() for specifiying significant digits
  • See also: floor(), ceiling()
• NOTE: To operate on more than one value, such as calculating a mean, your values need to be inside a vector.

What kinds of files will you use in RStudio? (1)

• R Notebook (.Rmd) or Quarto document (.qmd): mix formatted text and code and code outputs

• R script (.R): plain-text file that can be executed by R directly

  • ⚠ The only permitted “natural language” is in code comments.

• R Project (.Rproj): lives in the directory for a given project, and holds information like command history and settings. Optional but recommended.

• .RData: a workspace (session) snapshot

• .rds: an R data structure, i.e., an R object which has been saved to the filesystem

What kinds of files will you use in RStudio? (2)

Of course, you will also be loading files in whatever format your data take (spreadsheets, shapefiles, etc.).

Protip: make sure your operating system is set to display all file extensions!

• Windows instructions
• macOS instructions

Ways to run your session

• Console: quick calculations, one-line pieces of code
• Scripts: multiple console commands saved in one file
  • You can send to console line-by-line, or entire file at once
• Notebooks: a document with “code chunks” (mini scripts)
  • You can run the chunk one line at a time, or entire chunk at once

Working Directory (2)

• Use getwd() to check your current working directory
• To set your working directory:
  • In the Files tab (bottom right), click the three dots … at right, find the folder you would like to use as Working Directory, and click Open
  • Now click the ⚙More▾ button and choose Set As Working Directory
• When you run a script, you should set the working directory
• When you open a notebook, RStudio will automatically treat the notebook’s location as the working directory

Install a package

1. Google “Rstats \(x\)” where \(x\) is whatever you want to do (or maybe “R package for \(x\)”). Read about available packages for your job and choose one.
2. Go to Packages tab (bottom right pane ▛) and click Install.
3. Type the name of the package you want and click install.

Or if you like to write code: install.packages("name-of-package")

Let’s install the tidyverse, a collection of packages that we’ll use for the rest of the course: install.packages("tidyverse")

Attach a package

• To use a package, we need to attach it:
  • In Packages tab, click the checkbox next to the package name
  • or run library(package-name)
• Once a package is attached, its functions and/or datasets are available in the workspace. (But all packages are detached when the session ends.)
• Some packages will display a message or warning when they attach; some will not; depending on the package and your setup

Now let’s attach tidyverse: library(tidyverse)

Tabular data

You are likely to encounter tabular data in the following storage formats:

• Comma-separated values (CSV) or tab-separated values (TSV): .csv, .tsv, .dat, .txt
• Excel spreadsheets: .xlsx, .xls
• OpenDocument spreadsheet from OpenOffice/LibreOffice: .ods
• Parquet, a columnar format which is very efficient: .parquet
• Relational databases: usually via remote connection
  • Requires writing structured query language (SQL) or using a SQL-based package.

We are going to focus on CSV, since it is a non-proprietary and extremely common format.

Loading a CSV

Object assignment

In order to do something with our data, besides look at them once, we need to tell R to assign the result of our expression—i.e., the output of read_csv()—to an object. We also sometimes call this storing or saving an object

We use a left-pointing arrow, <- (type less-than and hyphen) for assignment:

You may also use = (equals) for object assignment, although it is not recommended.

Viewing the data

Use the View() function on our loaded data to launch the Data Viewer, for example: View(my_values). This is the same as clicking the object’s name in the Environment pane.

You can also type the object’s name to see a brief textual representation of it, in the console or notebook.

A few more useful functions:

• dplyr::glimpse() or str() shows all columns listed top-to-bottom
• head() and tail() shows the top or bottom of the data frame
• 💡 summary() summarizes a vector (or each vector in a data frame) according to its data type

Data types

Every value in R has one of these types:

• numeric: real, decimal numbers
• integer: whole numbers
• character: text; should always be in quotation marks " " in code
• logical: TRUE and FALSE, also called Boolean
• complex: for imaginary values i.e. complex numbers (rare)
• raw: values are stored as bytes and not human-readable (rare)

Data structures

Regardless of type, every value is organized into a structure (usually with other values). These are the most common structures:

• Vector: we have already used this ordered collection of single type; R’s default structure; 1D
• List: ordered collection of varying types; 1D
• Data frame: a tabular structure; 2D
  • “table” and “data.table” are two alternatives to data frames with different use cases
• Matrix (2D), Array (\(n\)D): Used for linear algebra, and some machine learning algorithms operate on matrices
• A special vector is the factor, which allows only certain values (defined in the structure); used for categorical variables
• Model object: produced by a modeling function such as linear regression
• Many other specialized structures are offered by packages, built upon these components

The data frame

• A table in which each column (“variable”) is a vector of equal length
• Row \(n\) (or “case” or “observation” \(n\) ) is read by retrieving the \(n\)th value of each variable
• To reference a variable/column, use the format data_frame$var_name
• We’ll be spending the rest of our time in this course with data frames and vectors.
• (Under the hood, a data frame is a list with some added features.)